Generate minimum-Matomo compatibility checks into plugin test runs - #48

Description

Plugin CI already runs PluginTests against the minimum Matomo a plugin declares in plugin.json, but nothing in that suite compiles stylesheets or templates. A plugin can therefore use a core Less mixin or Twig function that only exists in a newer Matomo, pass every check, and fatal on every page for anyone running an older version. That is what happened in CustomReports (PG-5029): a .inDarkMode({...}) block shipped against require.matomo: ">=5.0.0-rc5" while the mixin only landed in Matomo 5.11.0, so the admin UI died with .inDarkMode is undefined in anonymous-file-0.less on 5.0–5.10.

This adds the missing coverage centrally rather than asking every plugin repository to copy a test file in. Before running phpunit, the action writes small integration tests into the plugin checkout from templates in scripts/php/templates/:

• GeneratedAssetCompilationTest compiles the merged stylesheet with the plugin loaded, so a Less mixin or variable that does not exist in the Matomo under test fails here instead of in production.
• GeneratedTwigCompilationTest compiles every template the plugin ships, so a core Twig function, filter or tag that does not exist yet fails the same way. Plugins with no templates report a skip.

Because these land in plugins/<Plugin>/tests/Integration/, they run inside the existing minimum_required_matomo matrix leg — no new job and no extra CI minutes. Nothing changes in any plugin repository, and adding further checks later only requires another Generated*.php.tpl, which the generator picks up automatically.

A core test file could not do this: the min-version leg checks out an old core tag, so anything added to core today is simply absent exactly where the check is needed.

Scope of effect. The generation step is skipped entirely unless plugin-name is set, so Matomo core's own workflows are unaffected — they scope plugin work through ui-test-options and phpunit-test-options and never pass plugin-name. It is also skipped for suites that do not run plugin integration tests, when the plugin has no test directory, and when the new skip-generated-checks input is set.

Verified on real CI, not just locally. Six plugin workflows were pointed at this branch and dispatched: CustomReports, CustomAlerts, Funnels, FormAnalytics, HeatmapSessionRecording and TagManager — chosen for the largest Less and Twig surface. Every one of the ten minimum_required_matomo legs passed, each reporting Testing against '5.0.0-rc5' and running both generated checks against that core. TagManager's 884-test run reports no skips, so its 17 templates were genuinely compiled rather than skipped. The only red in the sweep was a known-flaky APITest in HeatmapSessionRecording, on a maximum_supported_matomo leg, unrelated to these checks.

That is six of roughly 46 plugins, so a plugin outside the sample could still be sitting on a real incompatibility. That is the check doing its job rather than a regression, and skip-generated-checks is the escape hatch if one needs to be unblocked while it is fixed.

Two details worth knowing when editing the templates:

• The test class namespace is what makes the framework load the plugin. With a generic namespace the plugin is not loaded, the merge covers core only, and the check passes while testing nothing — so both templates assert isPluginLoaded() and fail loudly rather than silently.
• tests:check-direct-dependency-use greps plugin files for a vendor namespace preceded by a space, and most plugins assert an exact list of the files it matches. Generated code must never name a vendor namespace that way. The first pilot run failed on exactly this, and the templates now document the constraint.

Related: PG-5221.

Checklist

• [✔] I have understood, reviewed, and tested all AI outputs before use
• [✔] All AI instructions respect security, IP, and privacy rules

Review

• Functional review done
• Potential edge cases thought about (behaviour of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• Security review done
• Wording review done
• Code review done
• Tests were added if useful/possible
• Reviewed for breaking changes
• Developer changelog updated if needed
• New documentation added or existing documentation updated